Method for a sensor-based and memory-based representation of a surroundings, display device and vehicle having the display device

ABSTRACT

A method for a sensor-based and memory-based representation of a surroundings of a vehicle. The vehicle includes an imaging sensor for detecting the surroundings. The method includes: detecting a sequence of images; determining distance data on the basis of the detected images and/or of a distance sensor of the vehicle, the distance data comprising distances between the vehicle and objects in the surroundings of the vehicle; generating a three-dimensional structure of a surroundings model on the basis of the distance data; recognizing at least one object in the surroundings of the vehicle on the basis of the detected images, in particular by a neural network; loading a synthetic object model on the basis of the recognized object; adapting the generated three-dimensional structure of the surroundings model on the basis of the synthetic object model and on the basis of the distance data; and displaying the adapted surroundings model.

FIELD

The present invention relates to a method for a sensor-based andmemory-based representation of a surroundings of a vehicle, to a displaydevice for implementing the method, and to the vehicle having thedisplay device.

BACKGROUND INFORMATION

European Patent Application No. EP 1 462 762 A1 describes an environmentdetection device for a vehicle. The environment detection devicegenerates a virtual three-dimensional surroundings model and displaysthis model.

German Patent Application No. DE 10 2008 034 594 A1 relates to a methodfor informing an occupant of a vehicle, a representation of asurroundings of the vehicle being generated in the process.

U.S. Pat. No. 7,161,616 B1 describes a display of a synthetic image froma virtual observer perspective.

A camera of a vehicle is unable to detect areas behind objects in asurroundings of a vehicle. A representation of a rearward view ofanother vehicle detected by a camera from the front is not possible forthe driver for example. A surroundings model ascertained solely as afunction of recorded camera images, which is represented usually from aperspective at an angle from above, consequently displays thesurroundings typically in an incomplete manner. Furthermore, in the caseof tall and nearby objects, the conventional methods result in unnaturaldistortions in a displayed representation of the surroundings.

An object of the present invention is to improve the representation of asurroundings of a vehicle for a driver.

SUMMARY

The above objective may be achieved by example embodiments of thepresent invention.

The present invention relates to a method for a sensor-based andmemory-based representation of a surroundings of a vehicle, the vehiclehaving at least one imaging sensor for detecting the surroundings. Theimaging sensor preferably has a camera. In accordance with an exampleembodiment of the present invention, the method comprises a recording ofa series of images by the imaging sensor. Subsequently, distance dataare determined as a function of the recorded images, in particular atwo-dimensional depth map and/or a three-dimensional point cloud.Alternatively or additionally, the distance data, in particular thetwo-dimensional depth map and/or the three-dimensional point cloud, aredetermined as a function of distances between objects in thesurroundings and the vehicle detected by at least one distance sensor ofthe vehicle. The optional distance sensor comprises an ultrasonicsensor, radar sensor and/or lidar sensor. The distance data representthe detected and/or ascertained distances between the vehicle andobjects in the surroundings of the vehicle. The determination ofdistances from the vehicle or of the distance data using an activedistance sensor, for example using the lidar sensor and/or using theradar sensor and/or using ultrasonic sensors, has the fundamentaladvantage over a camera-based distance determination that distances arereliably detected even in poor lighting conditions and/or poor weatherconditions. There may be a provision for performing a selection of asensor type, e.g., camera and/or ultrasonic sensor and/or lidar sensorand/or radar sensor for determining the distance data as a function oflighting conditions and/or weather conditions and/or a vehicle speed.Subsequently, in a further method step, a three-dimensional structure ofa surroundings model is generated as a function of the determineddistance data, in particular of the depth map and/or the determinedpoint cloud. The structure of the surroundings model comprises inparticular a three-dimensional grid. Furthermore, at least one object inthe surroundings of the vehicle is recognized on the basis of thedetected images. The object is preferably recognized by a first neuralnetwork and/or by another type of artificial intelligence or anotherclassification method, for example by a support vector machine orboosted trees. For example, vehicles, pedestrians, infrastructureobjects, such as for example a traffic light, and/or buildings arerecognized as objects. Optionally, the first neural network additionallyascertains or recognizes an object class of the detected object and/oran object species of the detected object. There may be a provision forexample that a vehicle class of a subcompact car is recognized as theobject class and/or a manufacturer model of the subcompact car isrecognized as the object species. Thereupon, a synthetic object model isloaded from an electrical memory as a function of the detected object,the memory being situated for example within a control unit of thevehicle. Optionally, the object model is loaded additionally as afunction of the detected object class and/or the detected objectspecies. The synthetic object model may be a specific object model,which represents the detected object, or a generic object model. Thegeneric object model is parameterizable or is modified on the basis ofthe recognized object and/or of the recognized object class and/or ofthe detected object species and/or on the basis of the distance data.Thereupon, the generated three-dimensional structure of the surroundingsmodel is adapted on the basis of the loaded synthetic object model andof the distance data, the synthetic object model replacing or expandinga structural area of the generated surroundings model. In other words,the generated surroundings model is expanded by the loaded object modelson the basis of the detected and/or ascertained distances. Thissurroundings model adapted with the addition of the synthetic objectmodel is displayed to the driver, the display preferably occurring on adisplay screen of the vehicle and/or on a display screen of a mobileelectronic device. The method advantageously makes it possible to reduceunnatural distortions in a view of the surroundings model so that thedisplayed surroundings model appears more realistic and error-free.Furthermore, on account of the adaptation of the surroundings model bythe use of object models from the memory, areas in the surroundingsmodel that are not visible for a camera are represented realistically,for example a view of another vehicle not recorded by a camera. Onaccount of the method, the driver is moreover better and more quicklyable to estimate a driving situation, a distance from an object or aparking space, which additionally increases the driving comfort for thedriver.

In a preferred development of the present invention, an objectorientation of the recognized object is ascertained on the basis of thedetected images, the object orientation being recognized in particularby a second neural network and/or by another type of artificialintelligence or another classification method. In this development, thegenerated three-dimensional structure of the surroundings model isadapted additionally on the basis of the ascertained object orientation.As a result, the adaptation of the generated surroundings model isadvantageously performed more quickly and more reliably.

In a particularly preferred development of the present invention,segments or object instances are recognized in the surroundings of thevehicle on the basis of the detected images. The recognition of thesegments or object instances preferably occurs with the aid of a thirdneural network and/or another type of artificial intelligence or anotherclassification method. Subsequently, the distances or depth informationin the distance data are assigned to the recognized segment or therecognized object instance. In this development, the adaptation of thegenerated three-dimensional structure of the surroundings model occursadditionally on the basis of the recognized segments or objectinstances, in particular on the basis of the segments or objectinstances assigned to the distances. As a result, the adaptation of thegenerated three-dimensional structure of the surroundings model on thebasis of the loaded synthetic object model is advantageously performedmore precisely and more quickly.

In a further development of the present invention, a texture isascertained for the adapted three-dimensional structure of thesurroundings model on the basis of the detected images, the detectedimages preferably being camera images. Subsequently, the adaptedsurroundings model is displayed with the ascertained texture. Forexample, a color of a vehicle in the surroundings model is ascertained.There may be a provision for the ascertained texture to comprisedetected camera images or perspectively modified camera images. As aresult of this further development, the surroundings model is displayedwith realistic imaging and/or coloring, which facilitates easyorientation for the driver. Optionally, the ascertainment of the texturefor the structural area of the surroundings model adapted by the loadedobject model is loaded or ascertained from the memory, in particular onthe basis of the recognized object or a recognized object class or arecognized object species. For example, the texture of a manufacturermodel of a vehicle is loaded from the electronic memory if acorresponding object species was recognized. This development gives thesurroundings model a realistic appearance for the driver. Moreover, itprevents unnatural distortions in the texture.

In a preferred development of the present invention, the display of theadapted surroundings model occurs within a specified area around thevehicle. In other words, the surroundings model is represented onlywithin an area that is delimited by a specified distance around thevehicle. The specified distance around the vehicle is preferably smallerthan or equal to 200 meters, preferably 50 meters, in particular smallerthan or equal to 10 meters. The specified area is defined for example bya base area of the specified area, which represents the specifieddistance or the specified area. A center point of the base arearepresents in particular a center point of the vehicle. The base areamay have any shape, the base area preferably having a square, ellipticalor circular shape. This advantageously reduces the computing expenditurefor generating the surroundings model as well as the computingexpenditure for adapting the surroundings model. Furthermore, thisdevelopment advantageously achieves a low rate of unnatural distortionsin the texture of the displayed surroundings model.

There may be a provision for a size of the specified area and/or a shapeof the specified area and/or an observer perspective of the displayedsurroundings model to be adapted as a function of a vehicle speed and/ora steering angle of the vehicle and/or the distance data or as afunction of a detected distance from an object.

Another development of the present invention provides for the display ofat least one projection area outside of the adapted surroundings model,which is situated at least partially vertically with respect to a basearea of the specified area. At least one subarea of the currentlydetected image is projected onto this projection area, in particular asubarea of a detected camera image. The images represented on thedisplayed projection area represent a view of a distant surroundings,that is, of an area of the surroundings that lies at a distance outsideof the displayed surroundings model and further than the specifieddistance that delimits the specified area.

In a further development of the present invention, the projection areais displayed as a function of the vehicle speed and/or the steeringangle of the vehicle and/or the distance data or as a function of adetected distance from an object. This means that in the event of aparking process advantageously no far-range view is shown on theprojection area and consequently the computing expenditure is minimizedin this driving situation. Alternatively or additionally, a size and/ora shape of the projection area may be adapted as a function of thevehicle speed and/or of the steering angle of the vehicle and/or of thedistance data. Advantageously, this makes it possible, for example whendriving at a higher speed, to display to the driver a narrow section ofthe surrounding area lying ahead in the direction of travel, whichminimizes the computing expenditure in a driving situation at a higherspeed and focuses the attention of the driver to the area essential inthis driving situation.

The present invention also relates to a display device having a display,which is designed to carry out a method according to the presentinvention. The display device preferably has an imaging sensor, inparticular a camera, and a control unit. The control unit is designed tocarry out the method according to the present invention, that is to say,to detect the images, to generate and adapt the displayed surroundingsmodel and to control the display for displaying the adapted surroundingsmodel.

The present invention also relates to a vehicle comprising the displaydevice.

Further advantages are yielded by the description below of exemplaryembodiments with reference to the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 show a vehicle in accordance with an example embodiment of thepresent invention.

FIG. 2 shows a control unit in accordance with an example embodiment ofthe present invention.

FIG. 3 shows a flow chart of a method according to an example embodimentof the present invention.

FIG. 4 shows an image with recognized objects in accordance with anexample embodiment of the present invention.

FIG. 5 shows recognized object orientation on the basis of the imagefrom FIG. 4 in accordance with an example embodiment of the presentinvention.

FIG. 6 shows recognized segments on the basis of the image from FIG. 4in accordance with an example embodiment of the present invention.

FIG. 7 shows an example of a displayed surroundings model in accordancewith an example embodiment of the present invention.

FIG. 8 a base area of the specified area for representing thesurroundings model and projection area in accordance with an exampleembodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1 shows a top view of a vehicle 100 in accordance with an exampleembodiment of the present invention. Vehicle 100 has a forward-facingcamera 101 as an imaging sensor. Furthermore, wide-angle cameras 102 aresituated on vehicle 100 as imaging sensors at the front, at the rear aswell as on each side of the vehicle, which detect the surroundings 190of the vehicle. Vehicle 100 furthermore has distance sensors 103 and104, where the distance sensors may a lidar sensor 103 in this exemplaryembodiment, which may also be an imaging sensor, and multiple ultrasonicsensors 104, which may also be imaging sensors. Alternatively oradditionally, a radar sensor may also be situated on the vehicle, whichmay also be an imaging sensor. Lidar sensor 103 and ultrasonic sensors104 are designed to detect distances between vehicle 100 and objects 108a and 108 b in the surroundings of vehicle 100. Vehicle 100 moreover hasa control unit 105, which records the images detected by the camera andthe distances detected by lidar sensor 103 and/or ultrasonic sensors104. Control unit 105 is furthermore designed to control a display 106in vehicle 100 for displaying a visual representation of thesurroundings for the driver, in particular for displaying a surroundingsmodel generated and adapted by the control unit and, if indicated,projection areas outside of the surroundings model. To calculate thesurroundings model, control unit 105 loads data, in particular syntheticand/or generic object models, from an electrical memory 107 of vehicle100 and/or from an electrical memory of control unit 105.

FIG. 2 shows control unit 105 as a block diagram in accordance with anexample embodiment of the present invention. Control unit 105 detects atleast one sequence of images using camera 101 and/or optionally usingmultiple wide-angle cameras 102 and/or using lidar sensor 103.Furthermore, control unit 105 may optionally detect distances usinglidar sensor 103 and/or ultrasonic sensors 104. Control unit 105 isdesigned to load data from external memory 107 and/or from internalmemory 202 of the control unit. Processing unit 201 of control unit 105calculates a surroundings model on the basis of the detected imagesand/or the detected distances, which are combined in distance data, inparticular in a specific depth map and/or a specific point cloud, and/oron the basis of the data from memory 107 and/or 202. Control unit 105 isfurthermore designed to control a display 106 for displaying arepresentation of the surroundings of vehicle 100, the calculatedrepresentation being in particular the adapted surroundings model, itbeing possible to supplement the display with further information, forexample driving dynamics parameters, such as a vehicle speed, and/or aprojection area.

FIG. 3 shows by way of example a flow chart of a method in accordancewith an example embodiment of the present invention as a block diagram.The method begins with a detection 301 of a sequence of images using animaging sensor, in particular a camera 101 and/or 102. Optionally, in astep 302, distances between vehicle 100 and objects in the surroundingsof vehicle 100 are detected using at least one distance sensor 103and/or 104. Subsequently, in a step 303, distance data, in particular atwo-dimensional depth map and/or a three-dimensional point cloud, areascertained on the basis of the detected images and/or on the basis ofthe detected distances. Distance data, in particular the depth mapand/or the point cloud, comprise the detected or ascertained distancesof vehicle 100 from objects 108 a, 108 b in the surroundings of vehicle100. The distance data are ascertained for example on the basis of thedetected sequence of the images, in particular on the basis of anevaluation of an optical flow between detected camera images. Everydistance of the distance data or every point of the depth map and/or ofthe point cloud represents for example an ascertained distance betweenvehicle 100 and objects 108 a, 108 b in the surroundings of vehicle 100.Alternatively or additionally, step 303 may provide for the distancedata to be ascertained on the basis of images detected by a stereocamera. Alternatively, the distance data or the depth map and/or thepoint cloud are determined in step 303 on the basis of sensor systems101, 102, 103 and/or 104 that are independent of one another.Additionally, there may be a provision for the distance data or thedepth map and/or the point cloud to be ascertained on the basis of atime characteristic of data of a sensor system 101, 102, 103 and/or 104.For ascertaining the distance data, ultrasonic sensors 104 have forexample the specific advantage compared to a camera 101, 102 that thedetected distances are relatively independent of bad light and/orweather conditions. In a step 304, a three-dimensional structure of asurroundings model is generated on the basis of the distance data, inparticular of the depth map and/or the point cloud, thethree-dimensional structure comprising in particular a three-dimensionalgrid, the three-dimensional grid preferably simplifying or representingthe distance data. In an optional step 305, the areas of thesurroundings detected in an image are segmented based on a sequence ofthe detected images. For example, a segment or an object instance“roadway,” a segment “object,” a segment “building” and/or a segment“infrastructure object” are recognized. In an optional step 306, therecognized segments or object instances are assigned to the distances orthe depth information in the distance data. In a step 307, at least oneobject in the surroundings of the vehicle is recognized on the basis ofthe detected images. This recognition is performed using a first neuralnetwork trained for this purpose. In a subsequent step 308, a syntheticobject model is loaded from memory 107 and/or 202 as a function of therecognized object. In an optional step 309, an object orientation of therecognized object is ascertained on the basis of the detected images,preferably by a second neural network. The object orientation mayrepresent a first approximation of the orientation of the object, acategory of a relative orientation of the recognized object with respectto the vehicle being ascertained for example from a set comprising thecategories “forward object orientation,” “object orientation toward theright” and/or “object orientation toward the left.” Thereupon, in afurther method step 310, the generated three-dimensional structure ofthe surroundings model is adapted as a function of the synthetic objectmodel and of the distance data, the synthetic object model replacing oradapting a structural area of the generated surroundings model. Theadaptation 310 of the generated three-dimensional structure of thesurroundings model may occur preferably additionally as a function ofthe ascertained object orientation. Subsequently, in an optional step311, a texture is ascertained for the adapted three-dimensionalstructure of the surroundings model on the basis of the detected images.An ascertainment 311 of the texture for adapted structural areas of thesurroundings model is not performed if a texture for this adaptedstructural area is loaded from the memory in an optional step 312. In afurther optional step 313, a shape of a specified area, a size of thespecified area and/or a display perspective of the adapted surroundingsmodel is adapted as a function of a vehicle speed, a steering angle ofthe vehicle, a detected distance between the vehicle and an objectand/or the current light conditions and/or the current weatherconditions and/or as a function of the selected sensor type forgenerating the distance data, the specified area being represented forexample by a base area. Subsequently, the adapted surroundings model isdisplayed 314, the ascertained and/or loaded texture being optionallydisplayed on the three-dimensional structure of the adapted surroundingsmodel. The display 314 of the adapted surroundings model occurs withinthe specified area or the base area of the specified area around vehicle100. In a further step 315, there may be a provision for displaying aprojection area outside the adapted surroundings mode, which is situatedat least partially vertically with respect to a base area of thespecified area, at least one subarea of a detected image, in particularof a camera image, being projected onto this projection area. A size anda shape of the projection area may be optionally adapted as a functionof the vehicle speed, of the steering angle and/or of the distance data.

FIG. 4 shows as the image a detected camera image of the forward-facingfront camera 101 of the vehicle including the objects 401, 402, 403, 404and 405 detected by step 307. The objects 401, 402, 403, 404 and 405 arerecognized by at least one first neural network trained for thispurpose. An object class may be recognized or assigned to the recognizedobjects, for example vehicle 401, 402 and 403, building 405 or tree 404.

FIG. 5 shows the object orientations 501 and 502, detected in step 309,of the recognized objects 401, 402 and 403 on the basis of the cameraimage shown in FIG. 4 by dashed lines for the category 501 “forwardobject orientation” and by a dotted line for the category 502 “rearwardobject orientation,” the object orientations 501 and 502 having beenrecognized by at least one second neural network trained for thispurpose.

FIG. 6 shows the segments or object instances 601, 602, 603, 605 and 606recognized in step 305 on the basis of the camera image shown in FIG. 4or a series of camera images, segments 601, 602, 603, 605 and 606 havingbeen recognized by at least one third neural network trained for thispurpose. Segment 601 represents for example an area in which the vehicleis able to drive. Segment 602 represents an object area and segment 603represents an area in which a vehicle is not able to drive. A greenspace area is represented by segment 605 and a sky area is representedby segment 606. The first neural network and/or the second neuralnetwork and/or the third neural network may be replaced by a moregeneral neural network or by a recognition method or by a classificationmethod or by an artificial intelligence, which recognizes objects,object orientations as well as segments.

FIG. 7 shows a displayed surroundings model 701. Vehicles wererecognized as objects in step 307, as a result of which the surroundingsmodel was respectively adapted by an object model 702 and 703. In otherwords, the object models 702 and 703 were inserted into surroundingsmodel 701 on the basis of the recognized objects, the recognized objectorientation and the recognized segments or the surroundings model wasadapted by the object models 702 and 703. Object model 701 accordinglyhas a structure adapted by two object models 702 and 703. In FIG. 7, theadapted surroundings model 701 is displayed only within a specifiedsquare area 704 around a center point of vehicle 705, vehicle 100 havingalso been inserted into the surroundings model as an additional objectmodel.

Additional projection areas 802 may be situated at the edge and outsideof the displayed surroundings model 701. A subarea of the detectedimages, in particular detected camera images may be displayed on theseprojection areas 802. The subareas of the images displayed on projectionareas 802 represent a distant view for a driver.

FIG. 8 shows a vehicle 100 including a specified area around thevehicle, which is represented by a base area 801 of the specified area,and a projection area 802. In this exemplary embodiment, the base area801 of the specified area is shown in perspective and is square.Alternatively, the shape of the base area could also be elliptical orcircular. Alternatively, the display may also occur from a perspectiveperpendicular from above or at an angle from the side. The shape of basearea 801 of the specified area and/or the length a and/or the width b ofbase area 801 of the specified area are adapted for example as afunction of the vehicle speed and/or of the weather conditions and/or ofthe visibility conditions, for example of a brightness or time of day.The adaptation of the length a and/or of the width b of base area 801 ofthe specified area is symbolized in FIG. 8 by arrows 803. Projectionarea 802 is curved in this exemplary embodiment and stands vertically orperpendicularly with respect to base area 801 of the specified area.Alternatively, projection area 802 may be situated as a non-curved planeon at least one side of base area 801 of the specified area, it beingpossible for example for a projection area 802 to be situated on eachside of base area 801. Furthermore, in another exemplary embodiment,projection area 802 may be situated around 360° and closed or as acylindrical lateral surface around base area 801 or around the specifiedarea. The length c and/or the height d of projection area 802 areadapted for example as a function of the vehicle speed. The adaptationof the length c and/or of the height d of projection area 802 issymbolized in FIG. 8 by arrows 804. Base area 801 is preferably alsodisplayed as part of the surroundings model, the base area beingascertained in particular on the basis of the detected images and/or ofthe distance data, in particular of the depth map and/or of the pointcloud, so that base area 801 reproduces for example areas of unevennessof a roadway.

1-10. (canceled)
 11. A method for a sensor-based and memory-basedrepresentation of a surroundings of a vehicle, the vehicle including atleast one imaging sensor configured to detect the surroundings, the atleast one imaging sensor including at least one camera, the methodcomprising the following steps: detecting a sequence of images, theimages being camera images; determining distance data based on thedetected images and/or based on a distance sensor of the vehicle, thedistance data including distances between the vehicle and objects in thesurroundings of the vehicle; generating a three-dimensional structure ofa surroundings model based on the distance data; recognizing, by aneural network, at least one object in the surroundings of the vehiclebased on the detected images; loading a synthetic object model based onthe recognized object; adapting the generated three-dimensionalstructure of the surroundings model based on the synthetic object modeland based on the distance data; and displaying the adapted surroundingsmodel.
 12. The method as recited in claim 11, further comprising thefollowing steps: ascertaining, by the neural network, an objectorientation of the recognized object based on the detected images; andadapting the generated three-dimensional structure of the surroundingsmodel additionally based on the ascertained object orientation.
 13. Themethod as recited in claim 11, further comprising the following steps:recognizing, by the neural network, an object instance in thesurroundings of the vehicle based on the detected images; assigning thedistances in the distance data to a recognized object instance; andadapting the generated three-dimensional structure of the surroundingsmodel additionally based on the object instance assigned to the distancedata.
 14. The method as recited in claim 11, further comprising thefollowing steps: ascertaining a texture for the adaptedthree-dimensional structure of the surroundings model based on thedetected images; and displaying the adapted surroundings model with theascertained texture.
 15. The method as recited in claim 11, wherein theadapted surroundings model displayed corresponds to a specified areaaround the vehicle.
 16. The method as recited in claim 15, wherein asize of the specified area and/or a shape of the specified area and/or adisplay perspective of the adapted surroundings model, is adapted basedon a vehicle speed and/or the distance data.
 17. The method as recitedin claim 15, further comprising: displaying a projection area outsidethe adapted surroundings model, which is situated at least partiallyperpendicularly with respect to a base area of the specified area, atleast one subarea of a detected image being projected onto theprojection area.
 18. The method as recited in claim 17, wherein thedisplay of the projection area occurs as a function of a vehicle speedand/or the distance data.
 19. A display device for a sensor-based andmemory-based representation of a surroundings of a vehicle, the vehicleincluding at least one imaging sensor configured to detect thesurroundings, the at least one imaging sensor including at least onecamera, the display device configured to: detect a sequence of images,the images being camera images; determine distance data based on thedetected images and/or based on a distance sensor of the vehicle, thedistance data including distances between the vehicle and objects in thesurroundings of the vehicle; generate a three-dimensional structure of asurroundings model based on the distance data; recognize, using a neuralnetwork, at least one object in the surroundings of the vehicle based onthe detected images; load a synthetic object model based on therecognized object; adapt the generated three-dimensional structure ofthe surroundings model based on the synthetic object model and based onthe distance data; and display the adapted surroundings model.
 20. Avehicle, comprising: at least one imaging sensor configured to detectsurroundings of the vehicle, the imaging sensor including at least onecamera; and a display device for a sensor-based and memory-basedrepresentation of the surroundings of the vehicle, the display deviceconfigured to: detect a sequence of images, the images being cameraimages; determine distance data based on the detected images and/orbased on a distance sensor of the vehicle, the distance data includingdistances between the vehicle and objects in the surroundings of thevehicle; generate a three-dimensional structure of a surroundings modelbased on the distance data; recognize, using a neural network, at leastone object in the surroundings of the vehicle based on the detectedimages; load a synthetic object model based on the recognized object;adapt the generated three-dimensional structure of the surroundingsmodel based on the synthetic object model and based on the distancedata; and display the adapted surroundings model.